September 30 + October 2, 2024
John Rauser of Pintrest (now Amazon), speaking at Strata + Hadoop 2014. https://blog.revolutionanalytics.com/2014/10/statistics-doesnt-have-to-be-that-hard.html
Choose a statistic that measures the effect
Construct the sampling distribution under \(H_0\)
Locate the observed statistic in the null sampling distribution
p-value is the probability of the observed data or more extreme if the null hypothesis is true
Choose a test statistic
Shuffle the data (force the null hypothesis to be true)
Create a null sampling distribution of the test statistic (under \(H_0\))
Find the observed test statistic on the null sampling distribution and compute the p-value (observed data or more extreme). The p-value can be one or two-sided.
NH.means <- NHANES |>
filter(!is.na(HealthGen) & !is.na(HHIncomeMid)) |>
group_by(HealthGen) |>
summarize(IncMean = mean(HHIncomeMid), count=n())
NH.means# A tibble: 5 × 3
HealthGen IncMean count
<fct> <dbl> <int>
1 Excellent 69354. 817
2 Vgood 65011. 2342
3 Good 55662. 2744
4 Fair 44194. 899
5 Poor 37027. 164Are the differences in means simply due to random chance??
Excellent Vgood Good Fair Poor
Excellent 0.000 4343.671 13691.99 25160.797 32326.906
Vgood -4343.671 0.000 9348.32 20817.126 27983.236
Good -13691.991 -9348.320 0.00 11468.806 18634.915
Fair -25160.797 -20817.126 -11468.81 0.000 7166.109
Poor -32326.906 -27983.236 -18634.92 -7166.109 0.000We can measure the overall differences as the amount of variability between each of the means and the overall mean:
\[F = \frac{\text{between-group variability}}{\text{within-group variability}}\] \[F = \frac{\sum_i n_i(\overline{X}_{i\cdot} - \overline{X})^2/(K-1)}{\sum_{ij} (X_{ij}-\overline{X}_{i\cdot})^2/(N-K)}\] \[SumSqBtwn = \sum_i n_i(\overline{X}_{i\cdot} - \overline{X})^2\]
# A tibble: 6,966 × 2
HHIncomeMid HealthGen
<int> <fct>
1 30000 Good
2 30000 Good
3 30000 Good
4 40000 Good
5 87500 Vgood
6 87500 Vgood
7 87500 Vgood
8 30000 Vgood
9 100000 Vgood
10 70000 Fair
# ℹ 6,956 more rows[1] 12148.175 7804.504 -1543.816 -13012.622 -20178.731[1] 147578150 60910286 2383368 169328332 407181201[1] 817 2342 2744 899 164[1] 120571348234 142651889943 6539963000 152226170649 66777716928\[SumSqBtwn = \sum_i n_i(\overline{X}_{i\cdot} - \overline{X})^2\]
NHANES |>
filter(!is.na(HealthGen)& !is.na(HHIncomeMid)) |>
mutate(IncomePerm = sample(HHIncomeMid, replace=FALSE)) |>
select(HealthGen, HHIncomeMid, IncomePerm) # A tibble: 6,966 × 3
HealthGen HHIncomeMid IncomePerm
<fct> <int> <int>
1 Good 30000 100000
2 Good 30000 2500
3 Good 30000 17500
4 Good 40000 17500
5 Vgood 87500 17500
6 Vgood 87500 100000
7 Vgood 87500 22500
8 Vgood 30000 100000
9 Vgood 100000 70000
10 Fair 70000 87500
# ℹ 6,956 more rowsreps <- 1000
SSB_perm_func <- function(.x){
NHANES |>
filter(!is.na(HealthGen)& !is.na(HHIncomeMid)) |>
mutate(IncomePerm = sample(HHIncomeMid, replace=FALSE)) |>
group_by(HealthGen) |>
summarize(IncMeanP = mean(IncomePerm), count=n()) |>
summarize(teststat = sum(count*(IncMeanP - GM)^2))
}
SSB_perm_val <- map(1:reps, SSB_perm_func) |>
list_rbind()
SSB_perm_val# A tibble: 1,000 × 1
teststat
<dbl>
1 15549069058.
2 21362573990.
3 12041124689.
4 15966876043.
5 18470463479.
6 17974162304.
7 14031080472.
8 18098237199.
9 12571653965.
10 13278612078.
# ℹ 990 more rowsIf the null hypothesis is true, the labels assigning groups are interchangeable with respect to the probability distribution.
typically,
\[H_0: F_1(x) = F_2(x)\]
(there are no distributional or parametric conditions)
Random Sample The concept of a p-value usually comes from the idea of taking a sample from a population and comparing it to a sampling distribution (from many many random samples).
Randomized Experiment The p-value represents the observed data compared to the treatment variable being allocated to the groups “by chance.”
The Economist, Sep 21, 2017
Innovative Higher Education, 40, pages 291–303 (2015).
Want to know if the population average score for the perceived gender is different.
\[H_0: \mu_{ID.Female} = \mu_{ID.Male}\]
Although for the permutation test, under the null hypothesis not only are the means of the population distributions the same, but the variance and all other aspects of the distributions across perceived gender.
Conceptually, there are two levels of randomization:
\(N_m\) students are randomly assigned to the male instructor and \(N_f\) are assigned to the female instructor.
Of the \(N_j\) assigned to instructor \(j\), \(N_{jm}\) are told that the instructor is male, and \(N_{jf}\) are told that the instructor is female for \(j=m,f\).
# A tibble: 4 × 3
# Groups: tagender [2]
tagender taidgender `n()`
<dbl> <dbl> <int>
1 0 0 11
2 0 1 12
3 1 0 13
4 1 1 11Stratified two-sample test:
macnell |>
group_by(tagender) |>
mutate(permTAID = sample(taidgender, replace=FALSE)) |>
select(overall, tagender, taidgender, permTAID) # A tibble: 47 × 4
# Groups: tagender [2]
overall tagender taidgender permTAID
<dbl> <dbl> <dbl> <dbl>
1 4 0 1 1
2 4 0 1 1
3 5 0 1 1
4 5 0 1 1
5 5 0 1 0
6 4 0 1 1
7 4 0 1 1
8 5 0 1 1
9 4 0 1 0
10 3 0 1 0
# ℹ 37 more rowsdiff_means_func <- function(.x){
macnell |> group_by(tagender) |>
mutate(permTAID = sample(taidgender, replace=FALSE)) |>
ungroup(tagender) |>
group_by(permTAID) |>
summarize(pmeans = mean(overall, na.rm=TRUE)) |>
summarize(diff_mean = diff(pmeans))
}
map(1:5, diff_means_func) |>
list_rbind()# A tibble: 5 × 1
diff_mean
<dbl>
1 -0.0909
2 -0.474
3 0.654
4 0.274
5 -0.00216permutation sampling distribution:
.pull-left[
]
.pull-right[
# A tibble: 1 × 1
p_val
<dbl>
1 0.048]
